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ABSTRACT 

Evaluation in its true concept should be a process 
for collecting information to make better decisions. The author 
discusses in detail four planning stages to evaluate programs. The 
first stage of the process is to ascertain the decision areas of 
concern. In the second stage, the evaluator must select the 
appropriate information-gathering instruments. In the third stage, 
the data must be collected and analyzed in advance of the decision 
maker's deadline. The final stage is to report the findings to the 
decision maker, in time for him to use them, and in a form he can 
understand. The author offers eight references on the subject of 
evaluation. (CA) 
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EVALUATION OF PROGRAMS 



No orte needs to be told today that the educational world, as well as 
the world In general. Is In a state of turmoil. The quiet solitude of 
academia Is the thing of the past. If It ever really existed. Society Is 
looking more and more to education to offer some rays of hope In the dark* 
ness of problems that loom larger and larger and despair of solution. Edu- 
cation Is responding with a bounty of new Ideas, methods, and programs. 

The educational Institution Is rare that Is not In torment over Its very 
place and purpose In today's complex world. 

The junior college In many respects is the most tormented segment of 
education. Reasons for this are manifold: Its open door policy, the enor- 

mous range of talents. Interests, and backgrounds of Its students, its 
multitude of course and curricula offerings — just to name a few. Con- 
comitantly, and for these very reasons, the junior college Is looked to by 
many to offer great hope for finding at least some partial solutions to the 
problems now being faced. Witness the amounts of money the urban colleges 
are obtaining through governmental and private grants. Witness this con- 
ference today. 

What I am here to discuss today is not the problems, not the grantsman- 
shlp, not the specifics of the programs being tried, not the reports re- 
quired by the funding agencies, although all these things are related to our 
discussion in some way. i am here simply to make a plea that you don't wait 
until your program is over to decide whether or not it's worth anything, 
but that you start, right from the Initial planning stages, to — if you 
will — evaluate your programs. 
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Evaluation Is a popular word today. It Is on the minds of many 
people. We hear talk of revision of grading procedures of students — 
the no "F" policy; we hear talk of students wanting to evaluate teachers; 
we are all, at least In California, concerned about attacks on our ten- 
ure system; we read requirements written Into grants; and there Is that 
word that Is beginning to haunt everybody — you saw It on the cover of 
the March Junior College Journal In two inch headlines — "Accountability." 
Yes, evaluation Is on people's minds. As we discuss It today, lets begin 
where we should begin, with a definition of terms. 

Just what do we mean by evaluation? My Merr iam-Webster tells me It 
means "finding the value of," or "appraisal." Very little help here. I 
suggest you think for a moment about what the word evaluation means to 
you In an educational context. I suspect that you would equate it with 
one or more of the following ideas: possibly observation, measurement 

or testing of some kind; equating actual performance with expected or 
hoped for performance; or possibly some sort of professional judgment 
I would concur that all these are aspects of evaluation but I submit 
that none Is adequate for a usable definition of evaluation that we can 
apply to college programs. The definition of evaluation that I would like 
to concentrate on today is one phrased by the people in the Teaching Re- 
search Division of the Oregon State System of higher education. Here it 
Is: 

Evaluation is a process of examining certain objects 
and events In the light of specified value standards 
for the purpose of making adaptive decisions. 
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Note right sway some obvious principles that follow from this de- 
finition. First, evaluation Is a process of gathering Information; second, 
the Information collected will be aimed toward Its use In aiding the de- 
cision maker; third. Information must be presented to the decision maker 
In a form that he can use l.t effectively; and fourth, different kinds of 
decisions may require different kinds of evaluation procedures. 

As viewed by the Center for the Study of Evaluation at UCLA, the 
process of evaluation consists of four stages. Their definition Includes 
these four stages, as follows: 

Evaluation Is the process of ascertaining the decision areas 
of concern, selecting appropriate Information, and collecting 
and analyzing Information In order to report summary data use- 
ful to decision-makers In selecting among alternatives. 

You will observe that the two definitions are quite similar. 

The key concept In the definition of evaluation I am proposing for 
your consideration today Is that the evaluator's function Is to provide 
to decision makers Information that can be used effectively to make deci- 
sions about alternative courses of action. 

Let me digress for a moment to comment on evaluation as opposed to 
research. Certainly research techniques are employed In the evaluation 
process, but to me the key difference Is the concept of the value standard. 
To a researcher, the prime concern Is a functional relationship — to dis- 
cover or explain some phenomena. This usually means he will design rather 
comprehensively his plan of action. The evaluator on the other hand Is 
concerned that better decisions will be made and he may revise his plan con- 
siderably as the project progresses. 

Let us consider now the first stage In the evaluation process — that 
of ascertaining the decision areas of concern. Thinking about this raises 
questions such as the following: What Is the purpose of the evaluation? 
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Who will make the decisions? What criteria will be used by the decision 
maker? What are the value standards he will use? Just what Is the pro- 
gram supposed to accomplish? Who decides what the program Is supposed to 
accomplish? And quite a few more that I am sure you could think of. 

All of this takes us back ultimately to the philosophical principles 
on which the college operates — what Is the nature of the good life? — 
what is Important? — principles such as: "everyone should have the op- 

portunity to become educated to the maximum of his capabilities and In- 
terests;" or, "the college should maintain an environment conducive to 
the development of programs which respond to the needs of students in a 
changing society." For more such principles read the opening pages of 
any college catalog. Or see the set of recommendations so eloquently 
prescribed by the staff of the Compensatory Education Project. 

These principles suggest the kinds of behavior patterns, the types if 
values and Ideals, and the habits and practices that the program will be 
aimed at, and from these philosophical principles are derived the goals of 
the program, usually in pretty general terms. These goals will guide the 
choice of activities to be Included In the program and from these goals 
should flow the specific objectives of the program, which In a real sense 
are the operational definitions of the goals. It Is certainly to be hoped 
that this hierarchy leading to specific objectives Is sensitive to the 
society the college serves, to the student to whom It directs Its efforts, 
and to the disciplines Involved. By this I mean clearly to recommend that 
In planning the program the community, the students and the faculty should 
have a voice. Let me suggest, in addition, that the program evaluator be 
Included In these Initial planning conferences, primarily to make sure that 
the objectives for the program finally agreed upon will be stated In a 
form amenable to evaluation. 

r* 
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Let me also suggest that in these early planning stages you include 
someone knowledgeable in the area of electronic data processing, primarily 
to make sure that the information to be collected will be collected in a 
form that will expedite its analysis. 

it is difficult to overemphasize the importance of good well-stated 
objectives. Some of the properties of a good well -stated objective are: 

(1) it should be defined clearly enough so that all involved in the pro- 
gram can recognize and understand it; (2) the activities necessary to its 
fulfillment are possible; (3) there should be serious intent to achieve 
it, even at considerable cost; and (4) there must be some way of de- 
termining, or at least estimating, the degree to which the objective is 
actually realized. This last point is probably the most important and at 
the same time the most difficult to accomplish. 

On the subject of statement of objectives, let me recommend to you 
Robert Hager's delightful little book on Preparing Instructional Objectives . 
If you haven't read it, you will find (t well worth your reading, not only 
for your ciasroom work but for preparing objectives for your programs. 

Another useful device is one attributed to C. F. Paulson of Oregon 
called the ABC D's of good objectives: A, you should consider the 
audience , describe your learners, what are their entry characteristics Into 
the program; B, behavior, what is the learner expected to do; C, conditions. 
what circumstances, givens and props provided, and 0, degree, what Is the 
criterion or by what do you determine whether or not the objectives have 
been met. Hager says it beautifully: "You should be able to find some 

way to evaluate anything you think important enough to spend a signifi- 
cant amount of time teaching. If you find something you feel sure you 
cannot measure the place to put effort is in trying vo find some way to 
measure it." And, we might add, to be sure you know what It Is you are 
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trying to measure, give considerable thought to the statement of your ob- 
jectives. 

Now, before we leave this first task of an evaluator, that of ascer- 
taining the decision areas of concern, we must Include another aspect. In 
ascertaining what the decision making body Is attempt ing:to do, the evalua- 
tor must know something of his theater of operation. In addition to Identi- 
fying the outcomes or objectives he must also obtain an adequate descrip- 
tion of the population to be studied and the criteria for their selection 
and an accurate description of staff, media, facilities, and planned activi- 
ties. We of course keep In mind that many of these things will be prelimin- 
ary and subject to change, but the evaluator should have Just as much 
Information as possible before he starts out on the remaining stages of the 
evaluation process. 

Let us now turn to the second area In which the evaluator must become 
Involved, that of selecting the appropriate information. -Now tha* we ' ave 
some Idea about what the program Is trying to accomplish and we know some- 
thing of the situational factors, the next question Is "what kind of in- 
formation do we want to collect and what instruments will help us get it?" 

There are several concerns here of course. First we might ask how much 
evaluation will be needed, or wanted, or will be able to be supplied. What 
kind of a budget is there for the evaluator? How much time will be allowed? 
How much help will he get? How much evaluation do the project directors 
really want? Do they want It badly enough to support it? Do they want 
what they really need? We must consider financial constraints — how much 
money Is available, how much of that money the administrators of the pro- 
gram will choose to spend on the evaluation. We may have to make some Im- 
portant decisions — for example, deciding between finding out information 
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from written questionnaire or by personal Interview. Personal Interview 
Is considerably more expensive than the written questionnaire but may pro- 
vide considerably more information. There are situational factors to be 
considered also. We must know something about the respondents. We must 
know what kind of knowledge they have about particular topics If we are 
going to ask them questions on a questionnaire for Instance. We must be 
able to know what amount of thought they will be able to give to these 
questionnaires. We must know something about their ability to communicate. 
Will they be able to answer the questions we are asking? There are many 
things to be considered before we just dive In asking questions or giving 
tests. 

Also, when we are beginning to think about what kind of instruments 
we are going to use to elicit information, let's be careful not to just 
select the obvious ones and the ones easy to get. I certainly cannot deny 
that one has to consider th*s selection problem in the light of all the con- 
straints on the evaluator, but maybe if you look around a little, you might 
be able to find something which won't cost you any more, which won't take 
any more of your time, but which will give you much more reliable and valid 
information. For example, its easy to settle for a grade point average as 
a measure of learning in a particular course but does it real lv give you 
an accurate measure of learning? Maybe it does, but I have my doubts in 
many instances. In any case, I'd like to consider some other type of 
measure, hopefully to get some kind of cross validation. As you'll see, 
this concept of cross validation is one I consider quite important. I 
think we should try as many approaches as possible. Don't settle for some 
pet idea or technique but consider other possible ways of looking at the 
situation, keeping in mind the objectives of the program. 
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Now, how do we go about finding Instruments that will help us attain 
the measurement of our objectives? 

There are many attributes of measuring Instruments that are Important, 
three In particular I think that we should look for In evaluating a pro- 
gram. First, the Instrument must have reliability, meaning that whatever 
the Instrument measures. It measures It consistently ~ It can be depended 
upon. Secondly, It must have some kind of validity — meaning It measures 
what It Is supposed to measure — It isn't measuring something else com- 
pletely different from what we want. The third one Is not always considered, 
but I think It is Important — especially when we are considering programs 
for the disadvantaged; the instruments should have some kind of relevance. 
This word Is abused and overused today, but tUiat I mean Is the Instrument 
should not be an affront In any way to the persons who are going to be asked 
to respond to It. 

Let us turn now to specifics: how do you find Instruments? 

I suggest first of all that you take a look at what standardized tests 
are available,. There are many advantages to using standardized tests. They 
have already been chebked for reliability and validity, they're generally 
less expensive and you can usually find a critical review In the Mental 
Measurements Yearbook, or elsewhere, in which some outside person tells you 
what he thinks of the test. There are disadvantages of course. One of the 
major ones Is that they are normed on groups which are generally not the same 
type of group that you are working with. And the validity coefficient they 
give you may be related to a criterion which Is not one you are concerned 
with. Also there are administrative problems — you must be sure to ad- 
minister the tests under conditions specified by the test publisher. But 
let me suggest that you at least look to see if you can find one. Now I 
think many times you are not going to be able to find one, so I am going 
to suggest some types of things you might do to make up one of your own. 
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Especially when you are trying to measure objectives In the affective area — 
where you want to learn something about attitudes of people — very seldom 
will you find a standardized Instrument that Is entirely appropriate to 
your specific situation. So why not try making one up? 

Let me mention three of the more popular and I think useful ways of 
constructing a home made attitude scale. 

First, consider the one that Is commonly known as the Thurstone scale. 
Here's generally the way one goes about It. You write some statements about 
whatever you are trying to get an attitude toward, making some of the state- 
ments very favorable, some of them very unfavorable, and some neutral. I 
suggest you make forty or fifty of these statements. Then, gtve them to 
somebody, not just one somebody, but several — as many as you can get, hope- 
fully 20 or more. Try to get those types of people who will be similar to 
the ones to whom you are going to administer this Instrument and ask them to 
rate each statement on a scale, usually chosen from zero to ten, as to 
whether or not they find It favorable or unfavorable. When this Is done you 
find the median scale value for each student; that Is, the middle value for 
alt these Judges and select Items for your final Instrument whose median 
scale values range as far as possible over the full scale. You may select 
10, 15, 20, or so for your Instrument. Next put them In a random order and 
then ask the respondent simply to check which ones he agrees with. His 
particular score on this Instrument will be the median of the scale values 
for the Items that he has checked, and you have a measure of his attitude to- 
ward the object under consideration. It's a very rough Instrument, as ail 
these home made things are, but It will give yoi some Idea of an Individual's 
attitude compared to those on which you have based what we might call norms. 
In other words, your Judges' responses. 
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Another and probably more popular technique ts the so called Likert 
scale. The Likert scale differs from the Thurstone In several respects. 
First, you write some statements, with about half favorable and half un- 
favorable. To each statement you attach a scale, usually of 5 to 7 points. 

On the 5 points, for example, you might use the words "strongly agree," 
"agree," "neutral or no opinion," "disagree," or "strongly disagree." After 
you have prepared your statements, you again give them to some Judges. Hope- 
fully this time you can find two groups, one which will react favorably and 
the other unfavorably. Then after you have selected on the basis of your 
Judges' response the Items which are favorable and unfavorable, you ad- 
minister these to your group, scoring the favorable Items with the values 
1, 2, 3, 4, 5 from strongly agree to strongly disagree; on the unfavorable 
reverse the sequence to 5t 4, 3, 2, 1. Then you simply get a total score 
for the individual. The Likert scale as I mentioned is probably the most 
popular of this type of thing and Is quite appropriate when you're Interested 
in some kind of relative index to compare one person with another, or to 
compare a pretest and posttest administered to the individual at the be- 
ginning of the program and later on after he has been subjected to the treat- 
ment of the program. 

Let me mention one other, the "semantic differential." This is a de- 
vice using a set of bipolar adjectives; for instance weak-strong, good-bad, 
important-unimportant. The person is asked to check a point on some kind of 
graphic scale between these two extremes indicating his reaction to the 
particular object that you are concerned with. This semantic differential is 
a relatively quick method of obtaining measures for several different ob- 
jects about which you want to measure attitudes. If you would like to go 
back to Osgood's original work on fehe semantic differential you can find some 
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particular sets of adjectives which will combine to give y?u a stronger 
measure of certain types of attitudes. If you do that however you are al- 
most back to the concept of the standardized test. What I'm suggesting Is 
that you make up your own, check It out with some people In advance and 
use It. For example, you might put "myself" at the top ar?d ask the re- 
spondent to Indicate hts feelings on several scales. Take a look at the 
difference in feelings about different persons' self -appraisal. You might 
use "my Instructors," "my text books," or a whole variety of concerns. 
Again, these are very rough measures and I'll have a difficult time de- 
fending them against an expert who might challenge them. Yet I believe 
they are useful if done carefully and cross validated with other types of 
measures. 

To recap then, the Thurstone Is a useful device to get some kind of 
an absolute measure of attitude. The Likert measure Is a very popular 
one useful for relative Indices comparing one person with another or pre- 
test with posttest and the semantic differential Is a quick method of ob- 
taining measures for a variety of objects. In all of these written home 
made Instruments let me caution you about one particular type of error 
that Is very often not thought about but Is easy to overcome If you do 
think about it — that Is the so-called "expert" error. You forget some- 
times when you're writing the statements that you are familiar with what's 
going on, you are familiar with a certain lingo or Jargon, or pedagese. 
Don't forget that maybe the person who is going to read It Isn't. Hake 
your statements and Instruction In as basic English as you can. Be sure 
that the person who Is going to read the Instrument will be able to under- 
stand It. 




You might think at this point, why all these fancy ways to measure at- 
titudes, why not design a questionnaire and just ask the person. And why not? 
A questionnaire can often be a very useful device, but let me point out two 
or three things about questionnaires that I think should concern you. In 
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the first place, be sure you know what you're trying to find out. Don't 
just ask questions for the sake of asking questions. I think It Is quite 
Important that every question Is put on a questionnaire with a certain ob- 
ject In mind. There are problems too with the collection and analysis of 
data, so think carefully about whether or not you're going to use direct 
short answers, multiple choice type questions or whether you're going to 
use open-end questions. Think ahead how you're going to categorize the In- 
formation you will get with open-ended questions. It might be difficult. 
Then too, I would suggest that when you start to make out a questionnaire 
you sit down and write at least 5 times as many questions as you think you 
are going to use. Then divide these up and try them on some people pre- 
ferably similar to the ones who will respond to your questionnaire. These 
might be students in the program, or the staff, or the community or whom- 
ever you can find. Information obtained from the questionnaire will be 
valuable if It has some object in mind and It can be reasonably obtained and 
is reliable. So check it out first, run some sort of a pilot and keep only 
those questions which you are sure everyone can understand and respond to. 

If you're going to use a mailed questionnaire for any reason, I suggest 
you consider the following 7 factors which have been known to affect the 
return. 

1. Who sponsors It? 

2. How attractive is it? 

3. How long Is it? 

k. What kind of cover letter goes with it? 

5. How easy is it to fill out and return? 

6. What inducements are there for it to be returned? 

7* Whp*: population do the respondents come from? 

Most people will tell you that if you get 50% return from a mailed 
questionnaire that you have a very good response. 25 to 30% is average. 

I find personally in dealing with students, graduates, and other follow-ups 
of people who are involved in certain programs that we have no problems in 
getting 60% or thereabouts providing you make the Instrument attractive, 
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short, and to the point, you enclose a self-addressed stamped return en- 
velope and you entice them In your cover letter by point out that their 
responses will help future students. 1 know of one study whlci> showed that 
use of attractive memorial stamps Increased the response rate significantly. 

What about Interviews? Interviews can give you much more In-depth In- 
formation than questionnaires. They are however, time-consuming and dif- 
ficult to perform. This a lot of people don't seem to realize. They think 
all you need to do Is go ask questions; but the order of the questions, the 
type of questions, the attitude of the Interviewer, his voice Inflection, 
the way he's dressed, the type of person he Is, all can have a tremendous 
effect on the responses, so while I think Interviewing Is a very fine tech- 
nique, I would advise you not to use It unless you think through carefully 
the various difficulties Involved and read some of the literature on how to 
conduct a good Interview. 

Another type of measure that Is very useful In some situations Is the 
so-called non-reactlve or unobtrusive measure. This Is In a sense the spy 
approach to find out about people without their knowing about It. 1 don't 
like to put It In the context of spying, but there are some things you can 
look at without bothering people and without Interfering with the program 
which will give you some useful Information. For example In an Art exhibit 
on campus It was found which particular Item of art was the most popular by 
simply looking at the wear on the rug. 

There are three general sources for these unobtrusive measures, one of 
which Is physical traces of past behavior as In the example which I mentioned. 
Another one Is archival records where you can find historical data about stu- 
dents, and the third Is plain and simple observation. We won't take time 
today to go into anymore particulars on this, but I might suggest a very nice 



Page 14. 

little book called Unobtrusive Measures written by Webb and several others 
that I suggest you look Into If you sre Interested In this approach to 
obtaining Information. 

Professor W. B. Michael and N. S. Metfessel of U.S.C. have tabulated 
a long list of criterion measures that might have some usefulness In evalua- 
ting school programs. Not all of them have value In the Junior college 
situation, but a good many of them do. Let me just read a partial listing 
In one of their five categories - this one listing "Indicators of status 
or change In student behaviors other than those measured by tests. Inven- 
tories, and observation scales In relation to the task of evaluating ob- 
jectives of school programs": 

absences, frequency of 
anecdotal records 

appointments, frequencies with which they are kept or broken 
attendance, frequency and duration 

books, numbers checked out of library, numbers reported read 
changes In program, frequency of occurrence 
choices expressed or carried out - vocational and educational 
citations - commendatory In both formal and Informal media of 
common I cat ton 

contacts - frequency or duration of between observed person 
and significant other person 
disciplinary actions taken - frequency and type 
dropouts 

elected positions 

extra curricular activities 

grade point average 

leisure activities 

1 tbrary card - possessed or not 

load - number of units 

peer group participation 

recommendations 

referrals 

skills - craft, P. E. and others not measured by available tests 
transfers 

I am sure that two or thaee people Involved In a college program could 
sit down for a couple of hours, do some blue-skying and come up with a list 
longer than this. Sure, some of the Ideas will be thrown out as worthless. 
Also sure, some useful criterion measures will emerge. 
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Let us now turn to the third area that I mentioned In the beginning 
that Is one of the responsibilities of the evaluator. After you have se- 
lected the appropriate Information tools you must collect and analyze the 
data. Now It Is pretty obvious at this point that I have very little time 
left to go Into a detailed discussion about collection and analysis of data. 
This Is unfortunate Jr a way because this area Is probably as Important as 
or more Important than some of the other aspects. However. I would like to 
limit my remarks to two or theee comments about this area today and let It 
go at that. 

First, the question, when do we collect the data? Keep In mind that 
the purpose of the evaluation Is to aid the deilslon maker. The Information 
will do the decision maker absolutely no good If It comes In today and the 
decision was yesterday. This of course means that the data must be collected 
well In advance of the decision maker's deadline. Hopefully the evaluator 
and the decision maker will be In constant contact so that the evaluator will 
know In time when this data must be collected. 

Now. what about analyzing the data when he gets Tt? I'm treating this 
aspect of the evaluation process very lightly for several reasons. One of 
course Is obvious — we can't do everything In one brief period. But an- 
other reason Is. I think sometimes we get so Involved with concern about 
statistical hypothesis tests, analysis of variance, multiple regression 
analysis, and a few of the other fancier statistical techniques, that we 
forget to look long and lovingly at the data and try -to get it In some kind 
of a broad perspective. Too often have I seen In comparing an experimental 
group and a control group the words "no significant difference" with no 
statement or apparent concern as to the Type TWo error involved In the 
significance test, and where the procedure description Indicates It could 
well be enormous. Too often have I seen lists of Items comparing two groups. 
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with statements "significant at the 5% level" indicated as having some 
meaning, with no apparent concern that, even if the data were collected and 
analyzed randomly, one item in twenty would show a significant difference. 

Too often have I seen a small correlation coefficient Indicated as signifi- 
cantly different from zero, yet no mention that it explains practically none 
of the variance. 

Sometimes I think we get so intrigued with the tool that we overtook 
the mission that the tool is designed to accomplish. Now this may be strange 
for me to be saying this, as I teach statistics. Yet I'm convinced that In 
most of the decisions involved in the evaluation process you don't need 
fancy statistical techniques and when you do need them you can find some ex- 
pertise on your campus to caii upon for assistance. I think that sophisticated 
statistical techniques are toots to be used in questionable decisions that 
are close. Quite often they are not close at aii — you can teii baaadiy from 
the data what the decision ought to be. 

Now that I have probably left the Impression that I think statistical 
techniques are useless, iet me hasten to correct that impression. For the 
decisions that are close, there is no substitute for the correct use of an 
appropriate statistical tool. We of course cannot discuss detailed statis- 
tical methodology today, but iet me just recommend to you what I have found 
to be a useful guide for selecting an appropriate tool I am referring to a 
table devised by. James Bea I rd of the Oregon Teaching Research Division and 
avaiiabie in their CORD Research Training Manual. Bealrd requires you to 
answer four questions, the answers to which, using his guide, iead you to an 

appropriate statistical tool. The four questions you must answer are: 

(1) what is your question — 

do you want to describe, compare, or relate? 

(2) how many samples (or variables) do you have? 

(3) are your samples independent? 

(4) what is the ievei of your data — nominal, ordinal or 

better than ordinal? 

For further information on this device I refer you to the CORD Manual. 
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And now just a brief word on the last area of the four that I mentioned 
that were importent In evaluation, that of reporting findings to the decision 
maker. I am sure that you are well aware that decisions are going to be 
made whether or not Information Is presented to the decision maker. It 
therefore behooves the evaluator to get the Information to the decision 
maker In time for him to make his decision. In a form that he can read and 
understand, and In such a way that the Information presented to him will 
make sense to him. Careful attention had better be paid to this If we ex- 
pect him to Implement the recommendations suggested by the evaluator. It's 
very Important, It seems to me, that the decision maker be able to scan the 
evaluator's report In a hurry and pick out the salient points, or chances 
are he's not going to bother with It. As some one has put It, If one has 
to search for a needle In a haystack It Isn't likely he'll be able to make 
a stitch In time. 

By Why of summary of what we have been saying, let me list for you 
eight steps In the evaluation process as seen by Professor Metfessel of 
U.S.C.: 

(1) Involve the total school community: lay, professional, 

student 

(2) Develop cohesive framework of broad goals and specific 
objectives 

(3) Translate the specific objectives Into planned courses 
of action 

(4) Select and/or construct Instrument* for furnishing measures 
allowing Inferences about program effectiveness 

(5) Periodically administer the Instrument 

(6) Analyze the data collected, using appropriate tools 

(7) Interpret the data according to judgmental standards 
or values 

(8) Hake recommendations, provide feedback to all Involved 

Hopefully, the program wit? be adapted or modified at this point 

and the cycle of the evaluation process starts once again. 

1 have been talking today as If all of you out there are evaluators. 

I suspect that In reality most of you are program directors or Involved In 
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programs In some staff position and that few of you are Involved whole- 
heartedly and completely In the evaluation process. If this Is true then 
I think I have said the right things today, because my objective Is to try 
to get those of you who are Involved In the program to be concerned about 
Its evaluation. This concept of evaluation that I've tried to put forth 
today (and I think It's the current one that most people are accepting) 

Is not one of somebody coming In from the outside looking over your shoulder 
threatening you, but an evaluation In Its true concept should be, as we've 
described It today, a process whereby Information Is collected to make bet- 
ter decisions. I hope that you who are Involved with planning these pro- 
grams (ard who are not already doing so) will consider allocating someone 
on your staff and part of your budget to evaluation. Relatively, It's 
quite inexpensive, and the rewards wilt be multifold. I hope that you will 
not fait Into the trap of equating program existence v.lth program effective- 
ness but that you wilt consider making evaluation part of your program, and 
assign an evaluator (or maybe a team of evaluators) right from the beginning 
to Initiate the process that we have tried to describe today. 

Thank you for the opportunity to participate in your conference. 
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